{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train a model using Watson Studio and deploy it in Watson Machine Learning\n",
    "\n",
    "This notebook will show how to use your annotated images from Cloud Annotations to train an Object Detection model using a Python Notebook in Watson Studio. After training and testing, some extra steps will show how to deploy this model in Watson Machine Learning as an online API. You can use this API from any application afterwards.\n",
    "\n",
    "As a suggestion you can use this dataset from Kaggle to test Cloud Annotation and this notebook: https://www.kaggle.com/issaisasank/guns-object-detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specify the credentials for the bucket you used in Cloud Annoations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# credentials = {\n",
    "#     'BUCKET': '$$$BUCKET$$$',\n",
    "#     'IBM_API_KEY_ID': '$$$IBM_API_KEY_ID$$$',\n",
    "#     'IAM_SERVICE_ID': '$$$IAM_SERVICE_ID$$$',\n",
    "#     'ENDPOINT': '$$$ENDPOINT$$$',\n",
    "# }\n",
    "credentials = {\n",
    "    'IAM_SERVICE_ID': 'iam-ServiceId-f0afd6e2-22d6-433a-91ce-4d02fab0a8e0',\n",
    "    'IBM_API_KEY_ID': 'Q5ZIqOmUOt9PB2lOZX4n1RzHUO-E_kYQ3RFhbSsEtjfm',\n",
    "    'ENDPOINT': 'https://s3.us.cloud-object-storage.appdomain.cloud',\n",
    "    'IBM_AUTH_ENDPOINT': 'https://iam.cloud.ibm.com/oidc/token',\n",
    "    'BUCKET': 'guns-object-detection'\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import shutil\n",
    "\n",
    "if os.path.exists('tmp') and os.path.isdir('tmp'):\n",
    "    shutil.rmtree('tmp')\n",
    "\n",
    "CLOUD_ANNOTATIONS_DATA = os.path.join('tmp', credentials['BUCKET'])\n",
    "\n",
    "os.makedirs(CLOUD_ANNOTATIONS_DATA, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import ibm_boto3\n",
    "from ibm_botocore.client import Config, ClientError\n",
    "\n",
    "def download_file_cos(local_file_name, key): \n",
    "    '''\n",
    "    Wrapper function to download a file from cloud object storage using the\n",
    "    credential dict provided and loading it into memory\n",
    "    '''\n",
    "    cos = ibm_boto3.client(\"s3\",\n",
    "        ibm_api_key_id=credentials['IBM_API_KEY_ID'],\n",
    "        ibm_service_instance_id=credentials['IBM_API_KEY_ID'],\n",
    "        config=Config(signature_version=\"oauth\"),\n",
    "        endpoint_url=credentials['ENDPOINT']\n",
    "    )\n",
    "    try:\n",
    "        res=cos.download_file(Bucket=credentials['BUCKET'], Key=key, Filename=local_file_name)\n",
    "    except Exception as e:\n",
    "        print('Exception', e)\n",
    "    else:\n",
    "        print('File Downloaded')\n",
    "        \n",
    "def get_annotations(): \n",
    "    cos = ibm_boto3.client(\"s3\",\n",
    "        ibm_api_key_id=credentials['IBM_API_KEY_ID'],\n",
    "        ibm_service_instance_id=credentials['IBM_API_KEY_ID'],\n",
    "        config=Config(signature_version=\"oauth\"),\n",
    "        endpoint_url=credentials['ENDPOINT']\n",
    "    )\n",
    "    try:\n",
    "        return json.loads(cos.get_object(Bucket=credentials['BUCKET'], Key='_annotations.json')['Body'].read())\n",
    "    except Exception as e:\n",
    "        print('Exception', e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotations = get_annotations()\n",
    "\n",
    "download_file_cos(os.path.join(CLOUD_ANNOTATIONS_DATA, '_annotations.json'), '_annotations.json')\n",
    "\n",
    "for image in annotations['annotations'].keys():\n",
    "    local_path = os.path.join(CLOUD_ANNOTATIONS_DATA, image)\n",
    "    download_file_cos(local_path, image)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_TRAIN_STEPS = 500\n",
    "MODEL_TYPE = 'ssd_mobilenet_v1_quantized_300x300_coco14_sync_2018_07_18'\n",
    "CONFIG_TYPE = 'ssd_mobilenet_v1_quantized_300x300_coco14_sync'\n",
    "\n",
    "import os\n",
    "CLOUD_ANNOTATIONS_MOUNT = os.path.join('tmp', credentials['BUCKET'])\n",
    "ANNOTATIONS_JSON_PATH   = os.path.join(CLOUD_ANNOTATIONS_MOUNT, '_annotations.json')\n",
    "\n",
    "CHECKPOINT_PATH = 'tmp/checkpoint'\n",
    "OUTPUT_PATH     = 'tmp/output'\n",
    "EXPORTED_PATH   = 'tmp/exported'\n",
    "DATA_PATH       = 'tmp/data'\n",
    "\n",
    "LABEL_MAP_PATH    = os.path.join(DATA_PATH, 'label_map.pbtxt')\n",
    "TRAIN_RECORD_PATH = os.path.join(DATA_PATH, 'train.record')\n",
    "VAL_RECORD_PATH   = os.path.join(DATA_PATH, 'val.record')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing dependencies\n",
    "\n",
    "In the next cell we will install the libraries that will be used. Since we are using an older version of Tensorflow and Numpy, compared to the version that is already installed by default in your environment. We highly suggest creating a custom environment in your Watson Studio project for this notebook, using the following configuration:\n",
    "\n",
    "``````\n",
    "# Modify the following content to add a software customization to an environment.\n",
    "# To remove an existing customization, delete the entire content and click Apply.\n",
    "# The customizations must follow the format of a conda environment yml file.\n",
    "\n",
    "# Add conda channels below defaults, indented by two spaces and a hyphen.\n",
    "channels:\n",
    "  - defaults\n",
    "\n",
    "# To add packages through conda or pip, remove the # on the following line.\n",
    "dependencies:\n",
    "\n",
    "# Add conda packages here, indented by two spaces and a hyphen.\n",
    "# Remove the # on the following line and replace sample package name with your package name:\n",
    "\n",
    "# Add pip packages here, indented by four spaces and a hyphen.\n",
    "# Remove the # on the following lines and replace sample package name with your package name.\n",
    "  - pip:\n",
    "    - numpy==1.19.5\n",
    "    - tensorflow==1.15.2\n",
    "``````\n",
    "\n",
    "Use Python 3.7 and any hardware configuration without CPU that you would like. This notebook was not prepared to support training using GPUs in Watson Studio. Use the next cell to install the other dependencies as normal. After creating the environment you will have to change it using the **Information** tab, on the right side menu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import pathlib\n",
    "\n",
    "# Clone the tensorflow models repository if it doesn't already exist\n",
    "if \"models\" in pathlib.Path.cwd().parts:\n",
    "    while \"models\" in pathlib.Path.cwd().parts:\n",
    "        os.chdir('..')\n",
    "elif not pathlib.Path('models').exists():\n",
    "    !git clone --depth 1 https://github.com/cloud-annotations/models\n",
    "\n",
    "# !pip uninstall Cython -y\n",
    "# !pip uninstall tf_slim -y\n",
    "# !pip uninstall opencv-python-headless -y\n",
    "# !pip uninstall lvis -y\n",
    "# !pip uninstall pycocotools -y\n",
    "# !pip uninstall numpy -y\n",
    "# !pip uninstall tensorflow -y    \n",
    "\n",
    "# !pip install numpy==1.19.5\n",
    "# !pip install tensorflow==1.15.2\n",
    "!pip install Cython\n",
    "!pip install tf_slim\n",
    "!pip install opencv-python-headless\n",
    "!pip install lvis --no-deps\n",
    "!pip install pycocotools\n",
    "\n",
    "%cd models/research\n",
    "!protoc object_detection/protos/*.proto --python_out=.\n",
    "\n",
    "pwd = os.getcwd()\n",
    "# we need to set both PYTHONPATH for shell scripts and sys.path for python cells\n",
    "sys.path.append(pwd)\n",
    "sys.path.append(os.path.join(pwd, 'slim'))\n",
    "if 'PYTHONPATH' in os.environ:\n",
    "    os.environ['PYTHONPATH'] += f':{pwd}:{pwd}/slim'\n",
    "else:\n",
    "    os.environ['PYTHONPATH'] = f':{pwd}:{pwd}/slim'\n",
    "%cd ../.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing Tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd models/research\n",
    "!python object_detection/builders/model_builder_tf1_test.py\n",
    "%cd ../.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate a Label Map\n",
    "\n",
    "One piece of data the Object Detection API needs is a label map protobuf. The label map associates an integer id to the text representation of the label. The ids are indexed by 1, meaning the first label will have an id of 1 not 0.\n",
    "\n",
    "Here is an example of what a label map looks like:\n",
    "\n",
    "````\n",
    "item {\n",
    "  id: 1\n",
    "  name: 'Cat'\n",
    "}\n",
    "\n",
    "item {\n",
    "  id: 2\n",
    "  name: 'Dog'\n",
    "}\n",
    "\n",
    "item {\n",
    "  id: 3\n",
    "  name: 'Gold Fish'\n",
    "}\n",
    "````\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "\n",
    "# Get a list of labels from the annotations.json\n",
    "labels = {}\n",
    "with open(ANNOTATIONS_JSON_PATH) as f:\n",
    "    annotations = json.load(f)\n",
    "    labels = annotations['labels']\n",
    "\n",
    "# Create a file named label_map.pbtxt\n",
    "os.makedirs(DATA_PATH, exist_ok=True)\n",
    "with open(LABEL_MAP_PATH, 'w') as f:\n",
    "  # Loop through all of the labels and write each label to the file with an id\n",
    "  for idx, label in enumerate(labels):\n",
    "    f.write('item {\\n')\n",
    "    f.write(\"\\tname: '{}'\\n\".format(label))\n",
    "    f.write('\\tid: {}\\n'.format(idx + 1)) # indexes must start at 1\n",
    "    f.write('}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate TFRecords\n",
    "\n",
    "The TensorFlow Object Detection API expects our data to be in the format of TFRecords.\n",
    "\n",
    "The TFRecord format is a collection of serialized feature dicts, one for each image, looking something like this:\n",
    "\n",
    "````\n",
    "{\n",
    "  'image/height': 1800,\n",
    "  'image/width': 2400,\n",
    "  'image/filename': 'image1.jpg',\n",
    "  'image/source_id': 'image1.jpg',\n",
    "  'image/encoded': ACTUAL_ENCODED_IMAGE_DATA_AS_BYTES,\n",
    "  'image/format': 'jpeg',\n",
    "  'image/object/bbox/xmin': [0.7255949630314233, 0.8845598428835489],\n",
    "  'image/object/bbox/xmax': [0.9695875693160814, 1.0000000000000000],\n",
    "  'image/object/bbox/ymin': [0.5820120073891626, 0.1829972290640394],\n",
    "  'image/object/bbox/ymax': [1.0000000000000000, 0.9662484605911330],\n",
    "  'image/object/class/text': (['Cat', 'Dog']),\n",
    "  'image/object/class/label': ([1, 2])\n",
    "}\n",
    "````\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import io\n",
    "import json\n",
    "import random\n",
    "\n",
    "import PIL.Image\n",
    "import tensorflow as tf\n",
    "\n",
    "from object_detection.utils import dataset_util\n",
    "from object_detection.utils import label_map_util\n",
    "\n",
    "def create_tf_record(images, annotations, label_map, image_path, output):\n",
    "  # Create a train.record TFRecord file.\n",
    "  with tf.python_io.TFRecordWriter(output) as writer:\n",
    "    # Loop through all the training examples.\n",
    "    for image_name in images:\n",
    "        try:\n",
    "            # Make sure the image is actually a file\n",
    "            img_path = os.path.join(image_path, image_name)   \n",
    "            if not os.path.isfile(img_path):\n",
    "                  continue\n",
    "\n",
    "            # Read in the image.\n",
    "            with tf.gfile.GFile(img_path, 'rb') as fid:\n",
    "                  encoded_jpg = fid.read()\n",
    "\n",
    "            # Open the image with PIL so we can check that it's a jpeg and get the image\n",
    "            # dimensions.\n",
    "            encoded_jpg_io = io.BytesIO(encoded_jpg)\n",
    "            image = PIL.Image.open(encoded_jpg_io)\n",
    "            if image.format != 'JPEG':\n",
    "                  raise ValueError('Image format not JPEG')\n",
    "\n",
    "            width, height = image.size\n",
    "\n",
    "            # Initialize all the arrays.\n",
    "            xmins = []\n",
    "            xmaxs = []\n",
    "            ymins = []\n",
    "            ymaxs = []\n",
    "            classes_text = []\n",
    "            classes = []\n",
    "\n",
    "            # The class text is the label name and the class is the id. If there are 3\n",
    "            # cats in the image and 1 dog, it may look something like this:\n",
    "            # classes_text = ['Cat', 'Cat', 'Dog', 'Cat']\n",
    "            # classes      = [  1  ,   1  ,   2  ,   1  ]\n",
    "\n",
    "            # For each image, loop through all the annotations and append their values.\n",
    "            for a in annotations[image_name]:\n",
    "                if (\"x\" in a and \"x2\" in a and \"y\" in a and \"y2\" in a):\n",
    "                    label = a['label']\n",
    "                    xmins.append(a[\"x\"])\n",
    "                    xmaxs.append(a[\"x2\"])\n",
    "                    ymins.append(a[\"y\"])\n",
    "                    ymaxs.append(a[\"y2\"])\n",
    "                    classes_text.append(label.encode(\"utf8\"))\n",
    "                    classes.append(label_map[label])\n",
    "\n",
    "            # Create the TFExample.\n",
    "            tf_example = tf.train.Example(features=tf.train.Features(feature={\n",
    "              'image/height': dataset_util.int64_feature(height),\n",
    "              'image/width': dataset_util.int64_feature(width),\n",
    "              'image/filename': dataset_util.bytes_feature(image_name.encode('utf8')),\n",
    "              'image/source_id': dataset_util.bytes_feature(image_name.encode('utf8')),\n",
    "              'image/encoded': dataset_util.bytes_feature(encoded_jpg),\n",
    "              'image/format': dataset_util.bytes_feature('jpeg'.encode('utf8')),\n",
    "              'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),\n",
    "              'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),\n",
    "              'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),\n",
    "              'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),\n",
    "              'image/object/class/text': dataset_util.bytes_list_feature(classes_text),\n",
    "              'image/object/class/label': dataset_util.int64_list_feature(classes),\n",
    "            }))\n",
    "            if tf_example:\n",
    "                # Write the TFExample to the TFRecord.\n",
    "                writer.write(tf_example.SerializeToString())\n",
    "        except ValueError:\n",
    "            print('Invalid example, ignoring.')\n",
    "            pass\n",
    "        except IOError:\n",
    "            print(\"Can't read example, ignoring.\")\n",
    "            pass\n",
    "\n",
    "with open(ANNOTATIONS_JSON_PATH) as f:\n",
    "    annotations = json.load(f)['annotations']\n",
    "    image_files = [image for image in annotations.keys()]\n",
    "    # Load the label map we created.\n",
    "    label_map = label_map_util.get_label_map_dict(LABEL_MAP_PATH)\n",
    "\n",
    "    random.seed(42)\n",
    "    random.shuffle(image_files)\n",
    "    num_train = int(0.7 * len(image_files))\n",
    "    train_examples = image_files[:num_train]\n",
    "    val_examples = image_files[num_train:]\n",
    "\n",
    "    create_tf_record(train_examples, annotations, label_map, CLOUD_ANNOTATIONS_MOUNT, TRAIN_RECORD_PATH)\n",
    "    create_tf_record(val_examples, annotations, label_map, CLOUD_ANNOTATIONS_MOUNT, VAL_RECORD_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download a base model\n",
    "\n",
    "Training a model from scratch can take days and tons of data. We can mitigate this by using a pretrained model checkpoint. Instead of starting from nothing, we can add to what was already learned with our own data.\n",
    "\n",
    "There are several pretrained model checkpoints that can be downloaded from the model zoo.\n",
    "\n",
    "The model we will be training is the SSD MobileNet architecture. SSD MobileNet models have a very small file size and can execute very quickly, compromising little accuracy, which makes it perfect for running in the browser. Additionally, we will be using quantization. When we say the model is quantized it means instead of using float32 as the datatype of our numbers we are using float16 or int8.\n",
    "\n",
    "````\n",
    "float32(PI) = 3.1415927 32 bits\n",
    "float16(PI) = 3.14 16 bits\n",
    "int8(PI) = 3 8 bits\n",
    "````\n",
    "\n",
    "We do this because it can cut our model size down by around a factor of 4! An unquantized version of SSD MobileNet that I trained was 22.3 MB, but the quantized version was 5.7 MB that's a ~75% reduction 🎉"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import tarfile\n",
    "\n",
    "import six.moves.urllib as urllib\n",
    "\n",
    "download_base = 'http://download.tensorflow.org/models/object_detection/'\n",
    "model = MODEL_TYPE + '.tar.gz'\n",
    "tmp = 'tmp/checkpoint.tar.gz'\n",
    "\n",
    "if not (os.path.exists(CHECKPOINT_PATH)):\n",
    "    # Download the checkpoint\n",
    "    opener = urllib.request.URLopener()\n",
    "    opener.retrieve(download_base + model, tmp)\n",
    "\n",
    "    # Extract all the `model.ckpt` files.\n",
    "    with tarfile.open(tmp) as tar:\n",
    "        for member in tar.getmembers():\n",
    "            member.name = os.path.basename(member.name)\n",
    "            if 'model.ckpt' in member.name:\n",
    "                tar.extract(member, path=CHECKPOINT_PATH)\n",
    "\n",
    "    os.remove(tmp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Config\n",
    "\n",
    "The final thing we need to do is inject our pipline with the amount of labels we have and where to find the label map, TFRecord and model checkpoint. We also need to change the the batch size, because the default batch size of 128 is too large for Colab to handle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#from google.protobuf import text_format\n",
    "\n",
    "from object_detection.utils import config_util\n",
    "from object_detection.utils import label_map_util\n",
    "\n",
    "pipeline_skeleton = 'models/research/object_detection/samples/configs/' + CONFIG_TYPE + '.config'\n",
    "configs = config_util.get_configs_from_pipeline_file(pipeline_skeleton)\n",
    "\n",
    "label_map = label_map_util.get_label_map_dict(LABEL_MAP_PATH)\n",
    "num_classes = len(label_map.keys())\n",
    "meta_arch = configs[\"model\"].WhichOneof(\"model\")\n",
    "\n",
    "override_dict = {\n",
    "  'model.{}.num_classes'.format(meta_arch): num_classes,\n",
    "  'train_config.batch_size': 24,\n",
    "  'train_input_path': TRAIN_RECORD_PATH,\n",
    "  'eval_input_path': VAL_RECORD_PATH,\n",
    "  'train_config.fine_tune_checkpoint': os.path.join(CHECKPOINT_PATH, 'model.ckpt'),\n",
    "  'label_map_path': LABEL_MAP_PATH\n",
    "}\n",
    "\n",
    "configs = config_util.merge_external_params_with_configs(configs, kwargs_dict=override_dict)\n",
    "pipeline_config = config_util.create_pipeline_proto_from_configs(configs)\n",
    "config_util.save_pipeline_config(pipeline_config, DATA_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Start training\n",
    "\n",
    "We can start a training run by calling the model_main script, passing:\n",
    "\n",
    "- The location of the pipepline.config we created\n",
    "- Where we want to save the model\n",
    "- How many steps we want to train the model (the longer you train, the more potential there is to learn)\n",
    "- The number of evaluation steps (or how often to test the model) gives us an idea of how well the model is doing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf $OUTPUT_PATH\n",
    "!python -m object_detection.model_main \\\n",
    "    --pipeline_config_path=$DATA_PATH/pipeline.config \\\n",
    "    --model_dir=$OUTPUT_PATH \\\n",
    "    --num_train_steps=$NUM_TRAIN_STEPS \\\n",
    "    --num_eval_steps=100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Export inference graph\n",
    "\n",
    "After your model has been trained, you might have a few checkpoints available. A checkpoint is usually emitted every 500 training steps. Each checkpoint is a snapshot of your model at that point in training. In the event that a long running training process crashes, you can pick up at the last checkpoint instead of starting from scratch.\n",
    "\n",
    "We need to export a checkpoint to a TensorFlow graph proto in order to actually use it. We use regex to find the checkpoint with the highest training step and export it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "import json\n",
    "\n",
    "from object_detection.utils.label_map_util import get_label_map_dict\n",
    "\n",
    "regex = re.compile(r\"model\\.ckpt-([0-9]+)\\.index\")\n",
    "numbers = [int(regex.search(f).group(1)) for f in os.listdir(OUTPUT_PATH) if regex.search(f)]\n",
    "TRAINED_CHECKPOINT_PREFIX = os.path.join(OUTPUT_PATH, 'model.ckpt-{}'.format(max(numbers)))\n",
    "\n",
    "print(f'Using {TRAINED_CHECKPOINT_PREFIX}')\n",
    "\n",
    "!rm -rf $EXPORTED_PATH\n",
    "!python -m object_detection.export_inference_graph \\\n",
    "    --pipeline_config_path=$DATA_PATH/pipeline.config \\\n",
    "    --trained_checkpoint_prefix=$TRAINED_CHECKPOINT_PREFIX \\\n",
    "    --output_directory=$EXPORTED_PATH\n",
    "\n",
    "label_map = get_label_map_dict(LABEL_MAP_PATH)\n",
    "label_array = [k for k in sorted(label_map, key=label_map.get)]\n",
    "\n",
    "with open(os.path.join(EXPORTED_PATH, 'labels.json'), 'w') as f:\n",
    "    json.dump(label_array, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating the results\n",
    "\n",
    "In the next steps we will use the images from the evaluation set to **visualize** the results of our model. If you don't see any boxes in your images, consider raising the amount of training steps in the **SETUP** section or adding more training images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import os\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "from PIL import Image as PImage\n",
    "from object_detection.utils import visualization_utils as vis_util\n",
    "from object_detection.utils import label_map_util\n",
    "\n",
    "# Load the labels\n",
    "category_index = label_map_util.create_category_index_from_labelmap(LABEL_MAP_PATH, use_display_name=True)\n",
    "\n",
    "# Load the model\n",
    "path_to_frozen_graph = os.path.join(EXPORTED_PATH, 'frozen_inference_graph.pb')\n",
    "detection_graph = tf.Graph()\n",
    "with detection_graph.as_default():\n",
    "    od_graph_def = tf.GraphDef()\n",
    "    with tf.gfile.GFile(path_to_frozen_graph, 'rb') as fid:\n",
    "        serialized_graph = fid.read()\n",
    "        od_graph_def.ParseFromString(serialized_graph)\n",
    "        tf.import_graph_def(od_graph_def, name='')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bbox_images = []\n",
    "for image_x in val_examples:\n",
    "    img_path = os.path.join(CLOUD_ANNOTATIONS_MOUNT, image_x)   \n",
    "    with detection_graph.as_default():\n",
    "        with tf.Session(graph=detection_graph) as sess:\n",
    "            # Definite input and output Tensors for detection_graph\n",
    "            image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')\n",
    "            # Each box represents a part of the image where a particular object was detected.\n",
    "            detection_boxes = detection_graph.get_tensor_by_name('detection_boxes:0')\n",
    "            # Each score represent how level of confidence for each of the objects.\n",
    "            # Score is shown on the result image, together with the class label.\n",
    "            detection_scores = detection_graph.get_tensor_by_name('detection_scores:0')\n",
    "            detection_classes = detection_graph.get_tensor_by_name('detection_classes:0')\n",
    "            num_detections = detection_graph.get_tensor_by_name('num_detections:0')\n",
    "            image = PImage.open(img_path)\n",
    "            # the array based representation of the image will be used later in order to prepare the\n",
    "            # result image with boxes and labels on it.\n",
    "            (im_width, im_height) = image.size\n",
    "            image_np = np.array(image.getdata()).reshape((im_height, im_width, 3)).astype(np.uint8)\n",
    "            # Expand dimensions since the model expects images to have shape: [1, None, None, 3]\n",
    "            image_np_expanded = np.expand_dims(image_np, axis=0)\n",
    "            # Actual detection.\n",
    "            (boxes, scores, classes, num) = sess.run(\n",
    "                [detection_boxes, detection_scores, detection_classes, num_detections],\n",
    "                feed_dict={image_tensor: image_np_expanded})\n",
    "            # Visualization of the results of a detection.\n",
    "            vis_util.visualize_boxes_and_labels_on_image_array(\n",
    "                image_np,\n",
    "                np.squeeze(boxes),\n",
    "                np.squeeze(classes).astype(np.int32),\n",
    "                np.squeeze(scores),\n",
    "                category_index,\n",
    "                use_normalized_coordinates=True,\n",
    "                line_thickness=8)\n",
    "            \n",
    "            bbox_images.append(image_np)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "fig = plt.figure(figsize=(50, 50))  # width, height in inches\n",
    "\n",
    "for i,bbox_image in enumerate(bbox_images):\n",
    "    sub = fig.add_subplot(len(bbox_images)+1, 1, i + 1)\n",
    "    sub.imshow(bbox_image, interpolation='nearest')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Here you can choose different images from the array to see it in more detail"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "plt.figure(figsize=(12, 8))\n",
    "plt.imshow(bbox_images[6])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploying your model in Watson Machine Leaning\n",
    "\n",
    "In the following steps we will export the artifacts that were created to a .tar file and upload the model to Watson Machine Learning. Than we will generate an online deployment using this model.\n",
    "\n",
    "You will need a Watson Machine Leaning instance and an IAM API Key in IBM Cloud that has access to this instance. See the steps in the documentation:\n",
    "\n",
    "https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-authentication.html\n",
    "\n",
    "Also, in the new version of WML you will need a Deployment Space and it's ID\n",
    "\n",
    "https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-spaces_local.html?audience=wdp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls $EXPORTED_PATH/saved_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -zcvf guns-object-detection-model.tar.gz -C $EXPORTED_PATH/saved_model ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watson_machine_learning import APIClient\n",
    "\n",
    "wml_credentials = {\n",
    "                   \"url\": \"https://us-south.ml.cloud.ibm.com\",\n",
    "                   \"apikey\":\"<apikey>\"\n",
    "                  }\n",
    "\n",
    "client = APIClient(wml_credentials)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.set.default_space(\"<deployment-space-id>\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.software_specifications.list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_spec = client.software_specifications.get_id_by_name('tensorflow_1.15-py3.6')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_meta = {\n",
    "    client.repository.ModelMetaNames.NAME              : \"Tensorflow Guns Object Detection Model\",\n",
    "    client.repository.ModelMetaNames.DESCRIPTION       : \"Guns Object Detection using Kaggle Dataset\",\n",
    "    client.repository.ModelMetaNames.TYPE              : \"tensorflow_1.15\",\n",
    "    client.repository.ModelMetaNames.SOFTWARE_SPEC_UID : model_spec\n",
    "}\n",
    "model_details_dir = client.repository.store_model( model=\"guns-object-detection-model.tar.gz\", meta_props=model_meta )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id_dir = model_details_dir[\"metadata\"]['id']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.hardware_specifications.list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta_props = {\n",
    "    client.deployments.ConfigurationMetaNames.NAME: \"Tensorflow Guns Object Detection Deployment\",\n",
    "    client.deployments.ConfigurationMetaNames.ONLINE: {},\n",
    "    client.deployments.ConfigurationMetaNames.HARDWARE_SPEC : { \"id\":  \"cf70f086-916d-4684-91a7-264c49c6d425\"}\n",
    "}\n",
    "deployment_details_dir = client.deployments.create(model_id_dir, meta_props )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment_id = deployment_details_dir['metadata']['id']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test the deployed model\n",
    "\n",
    "Choose one of the images from the evaluation set to score the model using the newly created API. This step can be done in another notebook or custom code, since your deployed model is not dependent of this kernel. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_path = os.path.join(CLOUD_ANNOTATIONS_MOUNT, val_examples[5])   \n",
    "if os.path.isfile(img_path):\n",
    "    print(\"OK\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image = PImage.open(img_path)\n",
    "# the array based representation of the image will be used later in order to prepare the\n",
    "# result image with boxes and labels on it.\n",
    "(im_width, im_height) = image.size\n",
    "image_np = np.array(image.getdata()).reshape((im_height, im_width, 3)).astype(np.uint8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = image_np.tolist()\n",
    "payload_scoring = {\n",
    "  \"input_data\": [{\n",
    "    \"values\": [data]\n",
    "  }]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "predictions = client.deployments.score(deployment_id, payload_scoring)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for x in predictions['predictions']:\n",
    "    if x['id'] == 'detection_scores':\n",
    "        scores = x['values'][0]\n",
    "    if x['id'] == 'detection_boxes':\n",
    "        boxes = x['values'][0]\n",
    "    if x['id'] == 'num_detections':\n",
    "        num = x['values'][0]\n",
    "    if x['id'] == 'detection_classes':\n",
    "        classes = x['values'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vis_util.visualize_boxes_and_labels_on_image_array(\n",
    "    image_np,\n",
    "    np.squeeze(boxes),\n",
    "    np.squeeze(classes).astype(np.int32),\n",
    "    np.squeeze(scores),\n",
    "    category_index,\n",
    "    use_normalized_coordinates=True,\n",
    "    line_thickness=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "plt.figure(figsize=(12, 8))\n",
    "plt.imshow(image_np)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}